Using optical codes with augmented reality displays

ABSTRACT

A technology is described for using a medical implement or a fluoroscopic image with reference to an image data set and a body of a person. A method may include detecting visual image data of a body of a patient and a medical implement. The optical codes on the body of the patient and on the medical implement may be identified. One operation is aligning the image data set with the body of the person using one or more optical codes on the body of the person and the fixed position of an image visible marker with respect to the optical code. A position of the medical implement with respect to the body of the person may be determined using one or more optical codes on the medical implement and the body of the person to reference the medical implement to the image data set and the body of the person.

BACKGROUND

Mixed or augmented reality is an area of computing technology whereimages from the physical world and virtual computing worlds may becombined into a mixed reality world. In mixed reality, people, places,and objects from the physical world and virtual worlds become a blendedenvironment. A mixed reality experience may be provided through existingcommercial or custom software along with the use of VR (virtual reality)or AR (augmented reality) headsets.

Augmented reality (AR) is an example of mixed reality where a livedirect view or an indirect view of a physical, real-world environment isaugmented or supplemented by computer-generated sensory input such assound, video, graphics or other data. Augmentation is performed as areal-world location is viewed and in context with environmentalelements. With the help of advanced AR technology (e.g. adding computervision and object recognition) the information about the surroundingreal world of the user becomes interactive and may be digitallymodified.

An issue faced by AR systems or AR headsets is identifying a positionand orientation of an object with a high degree of precision. Similarlyaligning the position of a virtual element with a live view of areal-world environment may be challenging. The alignment resolution ofan AR headset may be able to align a virtual object to a physical objectbeing viewed but the alignment resolution may only be aligned to withina few centimeters. Providing alignment to within a few centimeters maybe useful for entertainment and less demanding applications but greaterpositioning and alignment resolution for AR systems may be desired inthe scientific, engineering and medical disciplines. As a result,positioning and alignment processes may be done manually which can betime consuming, cumbersome, and inaccurate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example augmented reality (AR) environment inwhich a medical implement and image data of a patient may be referencedand aligned to actual views of the patient using one or more opticalcodes attached to the patient and the medical implement;

FIG. 2 illustrates an example of the optical code with an image visiblemarker that is affixed to a patient;

FIG. 3 illustrates an example of visual data that may be captured by acamera of an actual view a body of a patient and a medical implement andeach have an optical code affixed.

FIG. 4 illustrates an example of a view in an augmented reality (AR)display with annotations to guide positioning and orientation of amedical implement during a medical procedure.

FIG. 5 illustrates example of a view that may be displayed in anaugmented reality display device to display use or guidance informationfor a medical implement during a medical procedure.

FIG. 6 illustrates an example of using an augmented reality (AR) displayto enable positioning and orientation of a fluoroscopic image and animage projection from an image data set with respect to a body of aperson using optical codes.

FIG. 7 illustrates an example of a fluoroscopic device which is mobilewith respect to the body of the person and the fluoroscopic image andimage projection are also moved and/or modified.

FIG. 8A illustrates a fluoroscopic device, which is mobile with respectto the body of the person, generating a fluoroscopic image and an imageprojection in a coronal view for the fluoroscopic device to enablecombined viewing through an augmented reality (AR) headset or ARdisplay.

FIG. 8B illustrates an ultrasonic transducer used in combination withoptical codes to enable combined viewing of image data sets andultrasonic images through an augmented reality (AR) headset or ARdisplay.

FIG. 9a is a flowchart of an example method of using an augmentedreality headset to reference a medical implement with respect to animage data set and a body of a patient.

FIG. 9b illustrates a method for using an augmented reality (AR) displayto align a fluoroscopic image, an image projection from an image dataset and a body of a person using optical codes.

FIG. 10 illustrates an example system that may be employed in using anoptical code on a medical implement to enable the medical implement tobe referenced relative an image data set and to the body of a patient.

FIG. 11 is a block diagram illustrating an example of a computing systemto process the present technology.

DETAILED DESCRIPTION

A technology is provided for using an augmented reality (AR) headset toenable one or more optical codes to be identified on a medical implementthat is in view of a camera of the AR headset during a medicalprocedure. The medical implement can be referenced to an image data setthat is aligned with a body of a person using one or more optical codesand image visible markers on the body of the person. The image data setmay be a previously acquired image of a portion of body of a personusing a non-optical imaging modality (e.g., using MRI (magneticresonance imaging), CT (computed tomography) scanning, X-ray, etc.). Theimage data set can be aligned to the body of the person using an imagevisible marker that is a fixed distance from at least one optical codelocated on the body of the person. For example, an image visible markerand an optical code (e.g., an AprilTag or 2D optical bar code) may bothbe attached onto one piece of material (e.g., co-located or in fixedproximity of each other) to facilitate the alignment of the image dataset with the body of the person. An image visible marker is a markerthat can be viewed in a non-visible imaging modality, such as a capturedradiology image or an image data set, which may not be optically visibleto the AR headset. The image data set may be captured with arepresentation of the image visible marker using machine captured imagesthat capture structures of the human body with a non-optical imagingmodality. The representation of the image visible marker in the imagedata set may be aligned with the body of the patient using the knownfixed position of the image visible marker with respect to the one ormore optical codes affixed on the body of the person (as described infurther detail later). For example, the image visible marker may be aradiopaque marker, an MRI bead, an echogenic structure for ultrasound,etc.

Referencing the medical implement to the image data set may also includeidentifying a position and orientation of the medical implement withrespect to the body of the person and the image data set. Accordingly, amedical professional to view a virtual interior of the patient using theimage data set as referenced to a medical implement using an opticalcode on the medical implement, while looking at the actual patientthrough an AR headset. Visual image data that includes the medicalimplement may be captured using a visible light camera of the ARheadset. One or more optical codes that are visibly displayed on themedical implement may also be scanned. For example, the position andorientation of the medical implement may be determined by scanning theone or more optical codes (e.g., an APRIL code or a 2D (two dimensional)bar code). The medical implement may be a medical instrument, a trocar,a catheter, orthopedic hardware, a surgical implant, a clamp, anelectrocautery blade or system, an operating room object, an equipmentobject, a therapy object, a medical procedure object, a therapeuticobject, an insertable implement, an implantable object, a medicaldevice, etc.

A visual indicator, annotations, or a virtual tool may be integratedinto the image data set can be provided to guide positioning andorientation of the medical implement with respect to the body of apatient and the image data set using the AR headset. The medicalimplement or virtual tool may also have a graphical indicator (e.g.,computer generated graphics or symbols) displayed in proximity to or asan overlay to the medical implement or virtual tool using the AR headsetto highlight the medical implement or virtual tool, or the graphicalindicator may represent whether the medical implement is associated withthe medical procedure. Medical information can also be retrieved that isinstructional information describing the use of the medical implement ina medical procedure. In addition, the contours or outline of the medicalimplement may be detected using the one or more optical codes on themedical implement as a starting point. The use of automatic detectionand alignment may avoid or reduce time consuming and cumbersome manualalignment of the medical implement and image data set with actual viewsof the patient.

In another configuration of the technology, an augmented reality (AR)headset or an AR display can align and display a fluoroscopic image andan image projection from an image data set with respect to a body of aperson. A position and orientation of the fluoroscopic image as an imageprojection may be defined by a position and orientation of afluoroscopic device which is mobile with respect to the body of theperson. In this context, the description of an imaging device, which ismobile with respect to the body of the person, includes imaging devicemobility where the imaging device may change the imaging device'sorientation or move emitters, detectors, transducers, and/or imagingcomponents of the imaging device with respect to the body of the person.The one or more optical codes on the fluoroscopic device can be used todetermine the position and orientation of the fluoroscopic device withrespect to the body of the person or patient. The fluoroscopic image andthe image projection may be displayed with the portion of the body ofthe person being fluoroscopically imaged. This alignment and display canuse one or more optical codes and image visible markers on the body ofthe person and one or more optical codes on the fluoroscopic device.Optical codes on the body of the person and the optical codes on thefluoroscopic device can be identified in visual image data captured by acamera of an AR headset.

At least one of the optical codes on the body of the person can have afixed position relative to an image visible marker. This allows an imagedata set (e.g., a radiology image) to be aligned to the body of theperson using a fixed distance between the image visible marker and theone or more optical codes on the body of the person, as viewed throughan AR display (e.g., an AR headset). An image projection may be createdfrom the image data set based on the position and orientation of thefluoroscopic device. A fluoroscopic image from the fluoroscopic devicemay be aligned with the body of the person and the image projectionbased on the image visible marker (e.g., a radiopaque object) and/or theone or more optical codes defining the position and orientation of thefluoroscopic device. Further, the fluoroscopic image may be virtuallydisplayed in an AR headset in a location with respect to a body of aperson where the X-ray beam is passing through the body of the personand the fluoroscopic image may be aligned to overlay the portion of thebody of the person being imaged with the X-ray beam. The imageprojection may be oriented parallel to the fluoroscopic image and may bedisplayed in the AR headset as virtually being in at least part of apath of an X-ray beam. The aligned images may be displayed using the ARheadset along with the real world or real view of the patient or thealigned images may be displayed on a separate AR display (e.g., aseparate display screen). This process allows live fluoroscopic images,image data sets (e.g., augmented reality image or the image projection),and an actual view of a person to be combined, positioned, and orientedso that useful aspects of the fluoroscopic images (e.g., guiding ofradiopaque object within a body of a person) and the image data set(e.g., better tissue contrast, etc.) are combined during a medicalprocedure.

In another configuration using fluoroscopic images, a change in theposition and orientation of a fluoroscopic device can be detected withrespect to the body of the person using one or more optical codes on thefluoroscopic device. Then the image projection and fluoroscopic imageposition and orientation may be modified as defined by the change inposition and orientation of the fluoroscopic device. For example,movement of the projection of the image data set may be co-localized orsynchronized to match the fluoroscopic image based on a change inorientation and position of the fluoroscope device. The zooming of thefluoroscopic device may also be detected using a radiopaque object on abody of a person, and the size of the radiopaque object may be used toadjust a size of the image projection as viewed on an AR display.Graphical indicators, virtual tools, or a virtual targeting system mayalso be included on the image data set and co-localized to thefluoroscopic image to guide the positioning and orientation of afluoroscopically visible object (e.g., a trocar or needle) with respectto the body of the person and the image data set using the AR display.

In another configuration, the optical codes detected or captured by acamera or an AR headset may be used to confirm that the correct medicalprocedure is being performed on the correct patient. In addition,information related to the person or patient in the medical proceduremay also retrieved using the optical codes and may be displayed tomedical personnel using an AR system. The optical codes may also assistin confirming the identity of the patient. A confirmation may also beperformed to determine that a correct portion of the body and a correctmedical implement are in the medical procedure using one or more opticalcodes on the body and the medical implement.

FIG. 1 illustrates an example augmented reality (AR) environment 100 inwhich an image data set of a patient 106 or person may be aligned withactual views of the patient 106 using an optical code 200 affixed to thepatient 106. The environment 100 may include a physical space 102 (e.g.,operating theater, a lab, etc.), a user 104, the patient 106, multipleoptical codes 200 on the patient, a medical implement 118 with anoptical code 198, and an AR headset 108 in communication with a server112 over a computer network 110. A virtual user interface 114 and avirtual cursor 122 are also shown in dashed lines to indicate that thesevirtual elements are generated by the AR headset 108 and are viewable bythe user 104 through the AR headset 108.

The AR headset 108 may be an AR computing system that is capable ofaugmenting actual views of the patient 106 with an image data set. Forexample, the AR headset 108 may be employed by the user 104 in order toaugment actual views of the patient 106 with one or more 3D image dataset views or radiologic images of the patient 106 including, but notlimited to, bones 106 b (as illustrated in FIG. 1), muscles, organs, orfluids. The AR headset 108 may allow an image data set (or a projectionof the image data set) to be dynamically reconstructed. So, as the user104 moves around the patient 106, the sensors of the AR headset 108determine the location of the user 104 relative to the patient 106, andthe internal anatomy of the patient displayed using the image data setcan be reconstructed dynamically as the user chooses differentorientations relative to the patient. For example, the user 104 may walkaround the patient 106. Then the AR headset 108 may augment actual viewsof the patient 106 with one or more acquired radiology images or imagedata sets (MRI, CT scan, etc.) of the patient 106, so that both thepatient 106 and the image data set of the patient 106 may be viewed bythe user 104 from any angle (e.g., a projected image or a slice from theimage data set may also be displayed). The AR headset 108 may be amodified version of the Microsoft HOLOLENS, Meta Company META 2, EpsonMOVERIO, Garmin VARIA VISION or other AR headsets.

The optical code(s) 200 may be affixed to the patient 106 prior to thegeneration of image data of the patient 106 (e.g., capture of the MRI,CT scan, X-ray, etc.), and then remain affixed to the patient 106 whilethe patient 106 is being viewed by user 104 through the AR headset 108.Then, the optical code 200 and image visible marker may be employed bythe AR headset 108 to automatically align the image data set of thepatient 106 with actual views of the patient 106. Further, employing thesame optical code 200 used during the capturing of the image data toautomatically retrieve the image data may ensure that the image dataretrieved by the AR headset 108 matches the actual patient 106 beingviewed through the AR headset 108.

The AR headset 108 has sensor technology that may map or detect thesurface of the patient and similarly can map the surface of the medicalimplement 118, and this detected surface mapping data may beco-registered to the image data set. The medical implement 118 may befrequently moved in the environment 100, and the real-time position ofthe medical implement 118 may be tracked in the 3D space 102 using theoptical code and the medical implement 118 may be referenced to theimage data set 116 or a body of the patient 106. When the user 104inserts some portion of the medical implement 118 into the body of thepatient 106, the AR headset 108 may display a virtual inserted portionof the medical implement 118 projected into the image data set 116 todepict the medical implement 118 in the inner anatomy of the patient106. In this manner, the virtual inserted portion of the medicalimplement 118 may be projected onto actual views of the user 104 andreferenced to the image data set even when the actual inserted portionof the medical implement 118 is hidden from the actual views of the user104. The medical implement 118 may be tracked using one or more opticalcodes affixed to the medical implement 118, and then the one or moreoptical codes can be detected by the AR headset to establish acontinually updating position of the medical implement 118 withreference to the image data set 116 and the body of the person orpatient 106. In some embodiments, the medical implement 118 may beanything that the user 104 wishes to insert into the patient 106. Forexample, the medical implement 118 may include, but is not limited to, aneedle, a trocar, a scalpel (as illustrated in FIG. 1), a scope, adrill, a probe, a clamp, an implant, another medical instrument.

A virtual user interface 114 may be generated by the AR headset 108 andmay include options for altering the display of the projected inneranatomy of the patient 106 from the image data set 116 of the patient106. The virtual user interface 114 may include other information thatmay be useful to the user 104. For example, the virtual user interface114 may include information about the patient or the medical implements118 (e.g., medical instruments, implants, etc.) being identified with anoptical code. In another example, the virtual user interface 114 mayinclude medical charts or other medical data of the patient 106. In someconfigurations, the image data 116 or captured radiological data of aperson may be displayed by the AR headset 108 using a volume of theimage data 116 to display radiologically captured anatomy (e.g., bones106 b, tissue, vessels, fluids, etc.) of the patient 106 from the imagedata. This image data may contain axial slices, coronal slices, sagittalslices, or oblique slices of the image data. Slices may betwo-dimensional (2D) slices, three-dimensional (3D) slices, and/or fourdimensional (4D) slices (3D images with a time sequence of images) thathave a depth as well as a height and width (e.g., one or more layers ofvoxels). A user 104 may control the virtual user interface 114 using:hand gestures, voice commands, eye movements, remote controls (e.g., afinger clicker), a 3D mouse, a VR wand, finger sensors, haptictechnology, or other control methods.

In one example configuration, multiple users each wearing an AR headset108 may be simultaneously present to view the patient 106 augmented withimage data of the patient 106. For example, there may be multiple ARheadsets 108 that are used during medical procedures. One AR headset 108may be used by a first medical professional to adjust and manipulate theradiological images being displayed to both AR headsets and the secondhead set may be used by a second medical professional to assist inperforming the medical procedure on the patient. Additionally, onemedical professional may be able to turn on or off the radiologicalimage at the request of the other medical professional.

FIG. 2 illustrates the optical code 200 of FIG. 1 affixed to the patient106 of FIG. 1. With reference to both FIG. 1 and FIG. 2, the opticalcode 200 may be perceptible to an optical sensor, such as an opticalsensor built into the AR headset 108. In some embodiments, the opticalcode 200 may be an AprilTag, a linear barcode, a matrix two-dimensional(2D) barcode, a Quick Response (QR) code, or some combination thereof.An AprilTag is type of two-dimensional bar code which may be a visualfiducial system which is useful for augmented reality and cameracalibration. The AprilTags may be used to compute the 3D position,orientation, and identity of the tags relative to a camera, sensor, orAR headset.

The optical code 200 may be linked to medical data of the patient 106such that the medical data of the patient 106 can be accessed with theoptical code 200. For example, the optical code 200 may be used toautomatically retrieve the image data set to be used in a medicalprocedure for the patient using the AR system.

The optical code 200 may further be associated with markers 206 or imagevisible markers that are perceptible to a non-optical imaging modality.Examples of a non-optical imaging modality may include, but are notlimited to, an MRI modality, a Computerized Tomography (CT) scanmodality, an X-ray modality, a Positron Emission Tomography (PET)modality, an ultrasound modality, a fluorescence modality, an InfraredThermography (IRT) modality, 3D Mammography, or a Single-Photon EmissionComputed Tomography (SPECT) scan modality. In another example, thenon-optical images or image data sets may be an image or image data setwhich includes a combination of two or more forms of non-optical imagingmodality as listed above (e.g., two or more images combined together,combined segments of two or more non-optical images, a CT image fusedwith an MRI image, etc.). Each image data set in a separate modality mayhave an image visible code in the individual image data set which mayallow a PET image, a CT image, an MRI image, a fluoroscopy image, etc.,to be aligned and referenced together with an optical code on a body ofa person in an AR system view. Forming the markers 206 from a materialthat is perceptible to a non-optical imaging modality may enable themarkers 206 or image visible markers to appear in an image data set ofthe patient 106 that is captured using a non-optical imaging modality.Examples of markers 206 include, but are not limited to: metal spheres,liquid spheres, radiopaque plastic, metal impregnated rubber, metalstrips, paramagnetic material, and sections of metallic ink.

The markers 206 or image visible markers may be arranged in a patternand may have a fixed position relative to a position of the optical code200. For example, in the embodiment disclosed in FIG. 2, the opticalcode 200 may be printed on a material 202 (such as an adhesive bandage,paper, plastic, metal foil, etc.) and the markers 206 may be affixed tothe material 202 (e.g., embedded in the material 202 and not visible onany surface of the bandage). In this embodiment, the markers 206 may bearranged in a pattern that has a fixed position relative to a positionof the optical code 200 by being arranged in the fixed pattern in thebandage 202. Alternatively, the markers 206 may be embedded within theoptical code 200 itself, such as where the markers 206 are embeddedwithin an ink with which at least some portion the optical code 200 isprinted on the material 202 and the ink includes a material that isperceptible to the non-optical imaging modality, such as ink particlesthat are radiopaque and are not transparent to X-rays. In theseembodiments, the optical code 200 itself may serve both as an opticalcode and as the pattern of markers. Additionally, the markers 206 may bearranged by affixing or printing (at least temporarily) the optical code200 directly on the skin 106 a of the patient 106. By arranging themarkers 206 in a pattern that has a fixed position relative to aposition of the optical code 200, this fixed position may be employed tocalculate the location of the pattern of the markers 206 or imagevisible markers with respect to a visible location of the optical code200, even where the markers 206 are not themselves visible orperceptible to sensors of the AR headset 108.

Once the optical code 200 and the markers 206 are affixed to the patient106 in a fixed pattern, the non-optical imaging modality (to which themarkers 206 are perceptible) may be employed to capture image data ofthe patient 106 and of the markers 206. In particular, the image datamay include internal anatomy (such as bones 106 b, muscles, organs, orfluids) of the patient 106, as well as including the pattern of markers206 in a fixed position relative to the positions of the inner anatomyof the patient 106. In other words, not only will the internal anatomyof the patient 106 appear in the image data of the patient 106, but themarkers 206 will also appear in the image data set of the patient 106 ina fixed pattern, and the position of this fixed pattern of the markers206 will appear in the image data set in a fixed position relative tothe positions of the internal anatomy of the patient 106. In oneexample, where the non-optical imaging modality is a CT scan modality,the CT scan images may display the bones 106 b, organs, and soft tissuesof the patient 106, as well as the markers 206 arranged in a fixedposition with respect to the positions of the bones 106 b, organs, andsoft tissues of the patient 106.

Further, the patient 106 may be moved, for example, from a medicalimaging room in a hospital to an operating room in the hospital. Then auser 104 (such as a medical professional) may employ the AR headset 108to determine a location of the optical code 200 on the body of a personor patient. Next, the AR headset 108 may automatically retrieve theimage data of the patient 106 based on the optical code.

After detecting the optical code 200 in the 3D space 102, the AR headset108 may automatically calculate the position of the pattern of themarkers 206 in the 3D space 102 and with respect to one another. Thisautomatic calculation may be based on the sensed position of the opticalcode 200 in the 3D space 102 and may also be based on the known fixedposition of the pattern of the markers 206 relative to the position ofthe optical code 200. Even where the markers 206 are not perceptible tothe AR headset 108 (for example, due to the markers 206 being embeddedor underneath a material), the AR headset 108 can automaticallycalculate the location of the pattern of the markers 206 based on theposition of the optical code 200 and on the fixed position of thepattern of the markers 206 relative to the position of the optical code200. In this example, these fixed positions may enable the AR headset108 to automatically calculate the position of the pattern of themarkers 206 in the 3D space 102 with respect to one another even wherethe AR headset 108 is not directly sensing the positions of the markers206.

After calculating the location of the pattern of the markers 206 orimage visible markers in the 3D space 102, the AR headset 108 may thenregister the position of the internal anatomy of the patient 106 in the3D space 102 by aligning the calculated position of the pattern of themarkers 206 in the 3D space 102 with the position of the pattern of themarkers 206 in the image data set. The alignment may be performed basedon the calculated position of the pattern of the markers 206 in the 3Dspace 102 and the fixed position of the image data set to the markers206 relative to the positions of the internal anatomy of the patient106. This alignment and registration may then enable the AR headset 108to display in real-time the internal anatomy of the patient 106 from theimage data projected onto actual views of the patient 106.

Thus, the optical code 200, and the associated pattern of the markers206, may be employed by the AR headset 108 to automatically align theimage data of the patient 106 with actual views of the patient 106.Further, one or more optical codes 200 (e.g., an AprilTag and 2D barcode or another combination of optical codes) may be employed toautomatically retrieve the image data obtained during the capturing ofthe image data may ensure that the image data retrieved by the ARheadset 108 matches the actual patient 106 being viewed through the ARheadset 108.

In a further example, multiple optical codes 200 may be simultaneouslyaffixed to the patient 106 in order to further ensure accurate alignmentof image data of the patient 106 with actual views of the patient 106 inthe 3D space 102. Also, the pattern of five markers 206 disclosed inFIG. 2 may be replaced with another pattern, such as a pattern of threemarkers or a pattern of seven markers and each optical code may have adifferent pattern. Further, since the markers 206 are affixed to anoutside layer of the patient 106, the markers 206 may not all be in oneplane, but instead may conform to any curvatures of the outside layer ofthe patient 106. In these embodiments, the fixed position of the patternof the markers 206 relative to a position of the optical code 200 may beestablished after affixing the optical code 200 and the markers 206 tothe patient 106 to account for any curvatures on the outside layer ofthe patient 106.

FIG. 3 illustrates a visual image that may be captured by the camera ofthe AR headset or AR system. The visual image may include a body of apatient 300 and a medical implement 314, each with an optical codeaffixed. Optical codes may be used to identify the position andorientation of the medical implement 312 in the visual image and theoptical code may be used as a starting point identify the contour orshape of the medical implement 312. In addition, the optical code 310 onthe body of the person 300 may be used to access additional informationfor a patient (e.g., a patient record in a patient database) or avirtual object (e.g., in an object database) used as an overlay or as anadditional virtual object for display.

In one configuration, multiple optical codes 314 may be used on themedical implement 312 to enable the position and orientation of themedical implement to be determined. For example, an optical code 314 maybe affixed to multiple separate faces or surfaces of the medicalimplement. Each of these multiple optical codes may be coded to identifya specific face, aspect or orientation of the medical implement. Themultiple optical codes may also be used in identifying a desired 3D(three dimensional) virtual image to associate with the medicalimplement. For example, highlight graphics or a virtual object (e.g., avirtual medical instrument) may be selected as an overlay for themedical implement.

Once the optical codes 310, 314 affixed to the patient 300 and themedical implement 314 have been used to identify a position andorientation of the body of the patient and medical implement 314 withinthe visual image, this position and orientation information can betracked for each optical code and used when determining the position andorientation of the medical implements and patient in a procedure. Theoptical code 310 may be captured during the process of capturing avisual image of the patient during a procedure. As a result, the opticalcode 310 can be used to reference the medical implement 314 to thepreviously captured radiological images of the patient in an augmentedreality display (e.g., such as in an AR headset).

Furthermore, the optical code 314 on the medical implement can be usedto identify the particular type of medical implement 312 (e.g., amedical instrument or an orthopedic implant). Once the medical implementposition and orientation is identified 314, the position of the medicalimplement can be determined relative to the image data set discussedabove and the body of the patient. To better illustrate the position ofthe medical implement 314, the AR system may access informationdescribing the medical implement's size, shape, and contours and well asany other relevant information.

In one example, additional information associated with the medicalimplement 312 may also be retrieved. For example, information concerningthe use of a medical instrument may be retrieved and displayed in the ARheadset. This may include information regarding how to best align themedical instrument with body of the patient, tips for inserting animplant into a bone, settings for an electronic medical sensor, or otherguidance information for the medical implement.

In addition, the AR system may have records identifying which medicalimplements 312 are associated with a particular medical procedure. Byusing this information in combination with the optical codes, theaugmented reality system may determine whether a particular medicalimplement 312 is correctly associated with or is correctly being used inthe current medical procedure. For example, in a medical procedure inwhich a medical implement 312 is to be implanted in a patient, a datastore associated with the augmented reality system may be accessed toensure that the correct implant is being implanted in the patient andthat the correct tools are being employed. This combination of opticalcodes, procedure data, and patient data may be used to check: whether acorrect patient is being operated on, whether a correct body part isbeing operated on, whether an implant for the correct side of the bodyis being used or whether a correct implant size is being used. Use ofthe optical codes on the medical implements 312 prior to the procedure,may provide increased confidence that the correct medical implements 312are in the operating theater. In another example, the AR system maydisplay an optical indicator confirming that a given medical implement312 is authorized in a procedure or alerting a user that an incorrectmedical implement 312 or instrument is present.

FIG. 4 illustrates identifying additional information associated with amedical implement 312 during a medical procedure. As noted above, the ARsystem may use optical codes 314 to access information associated withthe patient 300, the medical implement 312, and/or the medical procedurebeing performed. The AR system can display information associated withthe use of a medical implement through an AR headset to aid a userduring the procedure. In one example, this information may describe theproper use of the medical instrument. For example, if the AR systemcaptures an optical code 314 on a medical instrumental 312, values inthe optical code 314 can be used as a lookup value to retrieveinformation for that instrument from a database. Such information mayfurther include planned medical procedure steps for use of the medicalinstrument 312 that can be displayed in the AR headset.

The medical professional may pre-plan certain aspects of a medicalprocedure. For example, the location and orientation of an incision or apath of cutting tissue may be pre-planned using annotations. These plansmay be entered into a database associated with the augmented realitysystem. When desired, the user can instruct the augmented reality systemto display the relevant pre-planned medical procedure information to theuser in the AR headset. In this case, the user or system may havepredetermined annotations 400, 402 to describe positioning of themedical instrument 612. A virtual guidance system may be displayed bythe AR headset to illustrate a pre-planned path or an alignment point,line or plane for a medical instrument for a medical procedure. As aresult, the augmented reality display may display a guidance annotation404, along with an indication to move the medical implement 312 to thecircle annotations 400, 402. When the medical professional has moved themedical instruments' position and orientation to the appropriatelocation and orientation (e.g., in three dimensions) as visuallydepicted using graphics in the AR headset, then graphical indicators,virtual tools, or a virtual targeting system may be displayed to showthat the proper position and orientation have been satisfied. Thepositioning and orientation of the medical implement 312 may be guidedin three dimensions or two dimensions. In one example, red indicatorsmay be displayed when the medical instrument is not aligned, and whenthe medical instrument is aligned then green indicators may be displayedthrough the AR headset. Thus, annotations may be modified, animatedand/or change color when a medical implement moves to defined positions,targets, entry points, or near target objects.

Similarly, a virtual implement or virtual tool may be displayed to allowalignment of a visible implement and virtual implement. For example, agraphical or virtual implement may be displayed or projected using an ARheadset. When the visible implement or real world implement is alignedwith the virtual implement or virtual tool, then the AR system candisplay a message or graphic indicating that the visible implement isaligned with the virtual implement. Further, this alignment of thevirtual implement may also enable alignments with viewable anatomicalstructures in a previously acquired image data set or an image data setacquired during the procedure (e.g., CT scans or Mill images obtainedduring the procedure).

FIG. 5 illustrates an incision point 504 is displayed in the AR displayas an overlay or virtual object on a specific part of the patient'sbody. The user can the use this virtual incision point to guide theinitial placement of the medical instrument with respect to the body ofthe person 300 and/or an image data set when beginning a surgery.Similarly, the ability to detect the position and orientation of themedical instrument may also include the ability to assist the medicalprofessional by guiding the depth or angle of an incision, properplacement of clamps, and so on. In addition, the system may displayinstructions for a correct angle or orientation to hold a given medicalimplement. For example, the AR display can present graphical indicators,virtual tools, or a virtual targeting system in the AR display forcorrect movement of the instrument in a current medical procedure basedon the position and orientation of the portion of the body of thepatient. For example, the system may display a visual indication of aline, along which an incision is to be made that is customized to anindividual body of a person. Similarly, the system can estimate acurrent depth of an instrument and indicate that the medical instrumentis to be moved with respect to the body of a patient and/or an imagedata set. During the procedure, the augmented reality system can use oneor more optical codes (e.g., either the optical codes on the patient 310or the medical implement 312) to access, align and display the planninginformation in the AR display to guide the medical professional.

In a medical preparation configuration, the use of the optical codes, asdescribed in this disclosure, may enable a medical professional who ispreparing a patient for a medical procedure to be graphically instructedto identify a portion of a patient's body or anatomy where the skinshould be prepped or other medical preparations should be made before amedical procedure. For example, after the AR headset has identified oneor more optical codes on patient's body, then graphical lines, graphicalshapes or virtual tools may be displayed in the AR headset to assist themedical professional performing medical preparation. This can guide themedical professional to position, orient, and prepare the patient in thecorrect anatomical location, prepare the correct incision point, and/orprepare on the correct portion of the patient's anatomy. For example, itmay be difficult to locate the correct vertebrae or hip location on apatient, and this technology provides such guidance. This guidance mayimprove the overall throughput in a surgical theater and for surgicaltechnicians.

FIG. 6 illustrates a technology 602 for using an augmented reality (AR)system to display a fluoroscopic image 654 and an image projection 656from an image data set, as aligned with respect to a body of a person606 a. A fluoroscopic device 660 may send a beam of X-rays (e.g.,continuous X-rays) through the patient to obtain a series offluoroscopic images or live video of the fluoroscopic imaging of thepatient viewable by the medical professional. The fluoroscopic device660 may also be mobile with respect to the body of the person. Thefluoroscopic image 654 and image projection 656 a may have positionand/or orientation defined by a fluoroscopic device 660 and/or an imagevisible marker. A camera or optical sensor (e.g., a visible light sensoror IR sensor) linked to the AR system or AR headset 608 may capturevisual image data of a portion of a body of a person on an operatingtable 603 and a fluoroscopic device 660 which is mobile with respect tothe body of the person. One or more optical codes on the body of theperson 606 a and one or more optical codes on the fluoroscopic device652, 658, can be identified and scanned in from the visual image datausing optical code recognition and scanning techniques or other opticalcode recognition techniques.

An image projection 656 may be selected to display a portion an imagedata set that is associated with (e.g., parallel to, oblique to, oranother fixed orientation) a fluoroscopic image 654 is being captured.The image projection may also display a specific anatomy type for thebody of the person, such as veins, nerves or bones. The fluoroscopicimage 654 may be a single layer projection (e.g., a two-dimensional (2D)projection). Alignment, positioning and orientation may be performedusing: optical codes 620 and image visible markers on the body of theperson, representations of image visible markers on the imageprojection, image visible markers captured in the fluoroscopic image654, and optical codes 652, 658 on the fluoroscopic device (e.g., aC-arm device, a catheterization lab, an angiographic lab, or afluoroscopic device with a movable emitter and detector).

At least one of the optical codes 620 on the body of the person can havea fixed position relative to an image visible marker (as described indetail previously). This allows an image data set (e.g., a capturedradiology image) to be aligned with the body of the person 606 using thefixed position of the image visible marker with reference to the one ormore optical codes on the body of the person. The body of the person maybe covered with fabric 607 but the internal anatomy 650 of the body maybe virtually viewed using the image data set.

An image projection 656 may be created from the image data set based onthe position and orientation of the fluoroscopic device 660. The imageprojection 656 may be a coronal projection, an oblique projection oranother projection orientation that matches the orientation of thefluoroscopic device 660. The image projection is projected from theimage data set to define a single plane view or a “slab” that is amulti-planar reconstruction (MPR) of the image data set (e.g., multiplelayer projection). The image projection may be of any selected thicknessand may be a MIP (maximum intensity projection) of tissue in theappropriate view.

A position and orientation of the fluoroscopic device 660 with respectto the body of the person 606 a can be determined using one or moreoptical codes 652, 658 on the fluoroscopic device 660. A fluoroscopicimage 654 from the fluoroscopic device 660 may be aligned with the bodyof the person and the image data set based on the image visible markerand/or the optical codes on a body of the person 606 a. Further, thefluoroscopic image 654 may be positioned and oriented using the positionand orientation of the fluoroscopic device 660 with respect to the bodyof the person 606.

A radiopaque marker may be used as the image visible marker to line upthe fluoroscopic image 654 with the body of the patient 606 a. In somesituations, the radiopaque marker may be the same image visible markerused for aligning the image data set or the radiopaque marker may be acompletely separate radiopaque marker (e.g., a lead rectangle) that mayhave a separate optical codes. For example, the radiopaque marker may afirst type of imaging modality marker used align the fluoroscopic image654 with a body of a person while a second imaging modality marker(e.g., a MRI type marker or an ultrasonic marker) may be used to alignan image data set with a body of a person. The aligned data set image,image projection, and fluoroscopic images may also be displayed usingthe AR headset 608 or on a separate AR display 662 along with the realworld view of the patient. Thus, the augmented reality images may becombined into the live fluoroscopic interventional or diagnosticprocedure performed on a body of a person 606 a.

Multiple useful views can be provided to a medical professional who isusing a fluoroscopic device and an AR system, as described earlier. Oneview may include the ability to take a partially transparentfluoroscopic image and overlay the fluoroscopic image so that thefluoroscopic image is anatomically aligned over the actual view of thepatient using the optical codes. Another view may enable an image dataset to be merged with the fluoroscopic image and be aligned with thepatient's body using the optical codes. Additionally, a projection fromthe image data set may move or be reconstructed in concert with themedical professional's changing actual view of the body through the ARsystem or AR headset. Yet another view may be provided in an AR systemthat displays a combined view of the fluoroscopic image and a projectionof the image data set (e.g., a 2D rectangular slice) that is parallel tothe fluoroscopic image as aligned with and overlaid on the patient(e.g., what the medical professional would see if the medicalprofessional were at the same perspective as the X-ray beam itself). Inthis configuration, the projection may move or be reconstructed as thefluoroscopic device moves.

A medical implement 618 with an optical code may also be referenced withrespect to the image data set and image projection 656, as describedearlier. This may enable a medical professional to view the medicalimplement 618 with reference to the image data set or image projection656 and a fluoroscopic image 654 simultaneously. The fluoroscopic image654 may be set to any level of transparency desired by the medicalprofessional or user.

FIG. 7 illustrates that a position of a fluoroscopic device 660 maychange with respect to a body 606 a of a person or a patient to enable amedical professional 604 to obtain a fluoroscopic image 654 from adifferent perspective. The change in the position and orientation of afluoroscopic device with respect to the body of the person can bedetected and quantified using one or more optical codes 652, 658captured by a camera associated with an AR headset 608 or AR system. Dueto the change in position and orientation of the fluoroscopic device660, the position and orientation of the image projection 656 andfluoroscopic image 654 may be modified. For example, the position and/ororientation of the image projection 656 may be moved, as viewed by theAR headset, based on detected changes in position and orientation of thefluoroscopic device 660 using the one or more optical codes 652, 658 ascompared to the body of a patient 606 a. The image projection 656 fromthe image data set may be reconstructed or a new projection may becreated using the modified position and orientation of the fluoroscopicdevice 660.

For example, the fluoroscopic device 660 may rotate 45 degrees in oneaxis. As a result, the image projection 656 may be recreated at that neworientation and the fluoroscopic image 654 may be displayed at therotated orientation, as viewed in the AR display, to enable theorientation of the fluoroscopic image 654 and the image projection 656to be aligned in the appropriate orientation with respect to the body ofthe person 606 a as viewed through AR headset. The fluoroscopic image654 may have a modified orientation in 3D space with respect to the bodyof the person as defined by the optical codes on the body, image visiblemarker, and/or as defined by the modified position and/or orientation ofthe fluoroscopic device 660. Thus, position and orientation of the imageprojection 656 and fluoroscopic image 654 changes when the position andorientation of the X-ray beam changes.

Determining the position and orientation of the fluoroscopic device 660relative to the patient, also enables the AR system to reconstruct theimage projection so the image projection 656 is parallel to thefluoroscopic image 654 obtained from the fluoroscopic detector. Inaddition, the fluoroscopic image 654 can be positioned with respect tothe body of the person 606 a based on the position where thefluoroscopic image 654 is being captured from the body (e.g., using theX-ray beam). Accordingly, the fluoroscopic image 654, the imageprojection 656, and patient's body 606 a may be aligned, so that amedical professional may see the anatomical structures of the person orpatient using the image projection 656 as an overlay to the fluoroscopicimage 654. The positioning and orientation of the image projection 656and fluoroscopic image 654 may represent an AR (augmented reality) viewbased on a portion of a body of a person the X-ray beam is passingthrough (as opposed to the point of view of the medical professional).

The AR system can re-construct the 3D image data set to provide aprojection from any angle that matches the position and orientation ofthe fluoroscopic device 654. For example, if fluoroscopic device 660 ispositioned to capture a lateral view then a lateral projection of theimage data set can be provided together with the lateral fluoroscopicimage view. Combining and aligning of the real world view of the body ofthe person, image projections from the image data set, and thefluoroscopic image enable a medical professional to better view andnavigate the internal anatomy of the patient. The 3D (three dimensional)image data set provides better soft tissue contrast (e.g., organs andblood vessels can be seen in the internal anatomy) and a 3D referenceenvironment that the fluoroscopic image may not provide.

In one configuration, the fluoroscopic device 660 may be zoomed withrespect to the body of the person 606 a while capturing the fluoroscopicimage 654. As a result, an adjustment to the image projection 656 may beperformed based on a change in size of an image visible marker (i.e., aradiopaque marker 830) captured in the fluoroscopic image 654 to enablethe image projection 656 to match a zoom of the fluoroscopic image 654.For example, if an image visible marker of a known size is captured(e.g., a lead marker of a defined size or length) when the fluoroscopicimage 654 is zoomed in, then the amount the image projection is to bezoomed in may be determined by the visual increase in size of the imagevisible marker in the fluoroscopic image (or decrease in the case ofzooming out). Alternatively, a zooming value may be electronicallyreported by the fluoroscopic device 660 or the zooming value may beprovided in or with a fluoroscopic image 654, and the image projection654 may be modified to match the fluoroscopic zoom, as electronicallyreported by the fluoroscopic device.

Further, as the magnification may be applied for the fluoroscopicdevice, there may be variations in magnification effects that may bechallenging to measure directly. For example, magnification variationsmay occur as distances between an X-ray detector and a subject change.Thus, the radiopaque marker can be used for zoom adjustments. So, if theradiopaque marker (e.g., an L shape) is identified in a zoomed in viewand the radiopaque marker is double the known physical size, then theimage data set can be scaled to match the doubled size of the radiopaquemarker. When the image data set is zoomed out or zoomed in, the imagedata set is not likely to align well with the actual view of the body ofthe person. As a result, the zoomed image data set and the fluoroscopicimage can be aligned and displayed in an AR display 662 that is separatefrom the AR headset or displayed in a portion of the AR headset field ofview that is not directly overlaid on the body of the person (e.g., offto the side of the center of the field of view of the AR headset).

To diminish magnification effects due to distances of patient anatomyfrom an X-ray source or X-ray detector, a round radiopaque sphere may bepositioned at the isocenter of the patient. When magnification is usedfor the fluoroscopic device, the magnification may be vary depending onthe distance of the body parts from the X-ray beam source. For example,the magnification of body parts that are closer to the X-ray beam sourcemay be greater. In order to correct for these differences inmagnification, the radiopaque marker used may be a metal sphere of knownsize (e.g., 1 cm). The metal sphere may be placed near the body part ofinterest (e.g., the isocenter of the image) and then the zoom of animage data set may be set to match the metal sphere. Thus, if the metalsphere appears smaller or larger in of the fluoroscopic image, the imagedata set can be zoomed based on that size. The metal sphere also appearsthe same from every angle. This appearance consistency enables the metalsphere to be an effective marker for detecting the zoom of thefluoroscopic image, which is to be applied to an image data set.

The use of an AR display 662 may also provide the ability to change aview of a fluoroscopic image and an image projection as aligned anddisplayed in an AR display in order to display the virtual patient tobetter match a position and orientation of the real view of a patient asdirectly viewed by a medical professional. Regardless of the orientationof the fluoroscopic image captured by the fluoroscope device, thedisplay of the aligned fluoroscopic image and image projection can beoriented in the AR display in a way that assists the doctor and/ormatches the actual view of patient's position and/or orientation. Forexample, the fluoroscopic image may be inverted either horizontally,vertically or oddly oriented in some other way as compared to a body ofpatient being actually viewed. A difficult viewing orientation may bedue to the capturing orientation of the fluoroscopic device 660.Accordingly, the image with the aligned fluoroscopic image 654 and imageprojection 656 can be flipped or reoriented (e.g., flipped horizontally,flipped by a certain number of degrees, moved to reverse the oddorientation) for viewing by the medical professional to make a medicalprocedure easier to perform or match more closely with what the medicalprofessional is seeing in an actual view. This ability to change animage orientation allows for a more intuitive interaction whenperforming procedures on a patient. For example, performing a medicalprocedure with fluoroscopic image guidance can be very difficult wheneverything is backwards.

Graphical indicators, virtual tools, or a virtual targeting system mayalso be used on the image data set or image projection to guide theposition and orientation of a fluoroscopically visible object (e.g., aneedle or a catheter) with respect to the body of the person and theimage data set, as viewed using the AR display. Similarly, graphicalindicators may be placed on the fluoroscopic image 654 to assist withguiding the fluoroscopically visible object during a medical procedure.Alternatively, the graphical indicators may be used to guide any medicalimplement 618 used in the fluoroscopic procedure.

FIG. 8A illustrates a side view of a combination of a cross-sectionalview of a body of a person 806, an image projection 804 from an alignedimage data set 820, an aligned fluoroscopic image 802 and a fluoroscopicdevice 814, which may enable a medical professional to fluoroscopicallyguide visible items 810 within a person's body 806. A fluoroscopicallyvisible object 810 (e.g., a needle) is able to be viewed and guided by amedical professional in the fluoroscopic image 802 with respect to theimage data set 820 and/or the image projection 804 aligned with the bodyof the person. The image projection 804 may be viewed in the AR headsetor AR display as being overlaid on the fluoroscopic image 802.Alternatively, the image projection 804 may appear to have thefluoroscopic image 802 overlaid on top of the image projection 804 orthe image projection 804 may appear to be within the aligned image dataset 820. As discussed earlier, graphical indicators, virtual tools, or avirtual targeting system can be provided on the image data set or imageprojection to guide the position and orientation of the fluoroscopicallyvisible object 810 with respect to the body of the person 806 and theimage data set 820.

The transparency of the fluoroscopic image 802 aligned with the imagedata set may be modified depending on the amount of the image projection804 or the real world body of the person 806 the medical professionaldesires to see. The fluoroscopically visible object 810 or medicalimplement may also have one or more optical codes on thefluoroscopically visible object 810 to be used to reference thefluoroscopically visible object 810 to the image projection 804. Inaddition, a position of the medical implement with respect to the bodyof the person and the image projection 804 or image data set 820 may bedetermined using optical codes on the medical implement and one or moreoptical codes on the body of the person 808, to enable the medicalimplement to be referenced to the image data and fluoroscopic image asviewed through the AR display.

The image projection 804 from an image data set 820 may have beencaptured using a Computed Tomography (CT) image or magnetic resonanceimage (MRI). Then the image projection 804 may be overlaid on a livefluoroscopic image 802. While a fluoroscopic image 802 is a live image,the fluoroscopic image does not have the 3D qualities or soft tissuecontrast resolution of an MRI (magnetic resonance image) or CT (computedtomography) image. Where the fluoroscopic image 802 has been referencedto a patient's body with one or more optical codes (e.g., an AprilTag)and the projection 804 of the previously acquired 3D image data set hasbeen referenced to an image visible tag on the patient's body, then amedical professional can view the virtual end of a needle as the tipmoves in the body of the patient 806 using the AR headset or AR display.This combines valuable aspects of the virtual images, the fluoroscopicimages, and a real view of a patient. A needle, a catheter tip orsimilar radiopaque object can be seen under fluoroscopy but the medicalprofessional cannot see certain soft tissue in the fluoroscopic image.Thus, the combination of the real view of a patient, an image data set,a fluoroscopic image, optical tags on the fluoroscopic device, imagevisible tags, a medical implement, and optical tags on a medicalimplement may allow the medical professional to see a medical implementwith reference to the image data set and/or the fluoroscopic image.

In another configuration, as illustrated in FIG. 8B, an ultrasound image860 may be obtained of a portion of a body of a patient 850 using anultrasound probe or ultrasonic transducer 862 while a medical procedureis being performed by a medical professional. The ultrasonic transducer862 may be mobile or movable with respect to the body of the person. Theultrasound image 860 or sonogram may be a 2D, 3D or a 4D (e.g.,including a time series) ultrasound image that is produced by soundwaves that bounce or echo off body tissues. The echoes are processed bya computer to create the ultrasound image or sonogram.

These ultrasound images 860 may be comparatively fast and inexpensive toobtain during a medical procedure but the resolution, accuracy andlocalization of the ultrasound images may not be as high as other typesof imaging such as CT scans, MRIs and other imaging modalities. Thistechnology provides the ability to combine the ultrasound image 860 withother image modalities to assist a medical professional in performing amedical procedure.

Accordingly, one or more optical codes 864 may be placed on or attachedto an ultrasound probe or ultrasonic transducer 862 (e.g., on theoutside housing). The optical code on the ultrasound transducer 862 maybe detected by the sensors in an AR headset. As a result, the positionand orientation of the ultrasonic transducer 862 may be detected and theposition and orientation of the ultrasound beam and the ultrasonicimages can be determined based on the position and orientation of theultrasonic transducer 862. In addition, one or more optical codes 852 ona patient or a body of a person may be detected to determine theposition and orientation of the patient 850. Knowing the position andorientation of the patient 850 and the ultrasound image 860 allows theultrasound image 860 to be aligned and projected, using the AR headset,onto a correct position on the patient. The ultrasound image 860projected onto the patient in the AR headset may be partiallytransparent (e.g., using a transparency value set by the medicalprofessional) or the ultrasound image may be opaque.

The ultrasound images 860 or a sonogram may also be combined with animage data set 866 (e.g., an Mill) which may be more accurate, clearer,higher resolution, larger, and have better tissue contrast informationas compared to ultrasound images 860. As described previously, theoptical codes 864 may be attached to an image visible marker 856. Thisimage visible marker 856 may allow the image data set 866 that has beenpreviously acquired from the patient to be aligned with the patient'sbody (as discussed earlier in detail). Thus, a medical professional canview the ultrasound images 860 combined with a high resolution imagedata set 866, as aligned and projected in the correct position on thepatient's body through the AR headset or AR system. For example, if amedical professional is performing a medical procedure on a patient'sliver using ultrasound equipment, a limited portion of the patient'sliver may be viewed at any one time using the ultrasound images but theentire liver may be viewed using the acquired image data set along withthe ultrasound images. The ultrasound image and image data set can beco-localized in the 3D space being viewed by the AR headset.

Typically, an ultrasound image 860 is not anchored to a reference pointrelative to the patient. Using the optical codes as described aboveprovides a reference point within a 3D space being viewed by the ARheadset. In addition, the reference points (i.e., optical codes) can beused to align the ultrasound images 860 with one or more image data sets866.

The ultrasonic transducer may use a fan beam or a linear beam that iselectronically guided. If the ultrasonic beam is moved or guidedmechanically or electronically, this guidance can be taken into accountwhen determining the location of the ultrasound image.

If a medical professional is going to perform a medical procedure suchas a breast biopsy, the medical professional may use ultrasoundequipment but it may be difficult to see the actual lesion in theultrasound image. However, the lesion may be visible in the CT, MRI,PET, nuclear, or other image data set that is displayed in combinationor co-localized with the ultrasound image. Thus, the ultrasound imagesmay be used to provide images captured (e.g., in real time) during theprocedure, while the previously captured detailed anatomy may besimultaneously referenced using a higher spatial or contrast resolutionimage data set.

The transducer may be passed over the surface of the body or insertedinto an opening of a body. This may allow the fused or composite viewsof the ultrasound images and the image data set to provide many varyingcomposite views of a body of a patient.

The alignment of the real time image with a body of a patient and imagedata sets, as described in FIG. 8B, may be applied to any type of realtime medical image where the position and orientation of the real timeimage may be obtained from a transducer, emitter or detector of theimaging device. An additional example of such real time imaging that maybe substituted for the ultrasound images is CT fluoroscopy images.

FIG. 9a is a flowchart of an example method of using an augmentedreality headset to co-localize a medical implement with respect to animage data set and a body of a patient during a medical procedure. Inthese and other embodiments, the method 900 may be performed by one ormore processors based on one or more computer-readable instructionsstored on one or more non-transitory computer-readable media.

The method may capture visual image data of a portion of a body of aperson or patient and a medical implement using an optical sensor of theAR headset, as in block 902. For example, a visible patient with anoptical code is registered or detected by a camera of the AR headsetused by a medical professional. One or more optical codes associatedwith the body of the person and the medical implement can be identifiedin the visual image data, as in block 904. An optical code on a body ofthe person may also be located in a fixed position relative to an imagevisible marker, as discussed earlier.

The image data set may be aligned with the body of the person using oneor more optical codes on the body of the person and the fixed positionof the image visible marker with respect to the optical code whenreferenced to a representation of the image visible marker in the imagedata set, as in block 906. In addition, a position and/or orientation ofthe medical implement with respect to the body of the person may bedetermined using one or more optical codes on the medical implement andthe body of the person to enable the medical implement to be referencedto body of the person and the image data set, as in block 908. Thealignment and merging of these multiple image aspects may be viewedthrough the AR headset. In one example, the image data set of radiologyimage can be presented at different levels of opacity depending on theneeds of the user or medical professional. In addition, this opacity maybe adjusted at any time.

In one configuration, the method may include providing a visualindicator, virtual guidance system, a virtual tool, a virtual highlight,or annotations on the image data set to guide positioning andorientation of the object with respect to the body of a patient usingthe AR headset. For example, the virtual tools or visual indicators mayinclude one or more of: a graphical highlight, 3D colorization, a 3Dsurgical tract or a virtual procedural track, a graphical highlight oftargeted anatomy, a graphical highlight of pathology, a graphicalhighlight of critical structures to avoid, or 3D visual code(s) andgraphical structures (e.g., lines, planes, cylinders, volumes,boundaries, or 3D shapes) in or around an anatomical structure may alsobe provided (e.g., to highlight an organ or mass in the body). Themethod may further include mapping a contour or 3D outline of themedical implement using an optical code as the starting reference point.Further, since the position and orientation of the medical implement isknown, a medical implement (e.g., a tool or implant) partially insertedinto a patient's body can still be monitored by the system to ensurecorrect placement and positioning with respect to the image data set andultimately the body of the person.

The method may include querying a database to determine whether themedical implement utilized or detected is assigned to be used in themedical procedure. In one example, the AR system may store medicalprocedure records indicating which patients are receiving specificmedical procedures. These records may include a list of specific medicalimplements (e.g., medical instruments, implants, etc.) that areassociated with individual procedures. If a medical implement isuniquely identified using an optical code, that optical code can be sentto the system and checked against the predetermined list of medicalimplements (each of which may be associated with separate opticalcodes). Thus, if an identified object is determined to be associatedwith the current medical procedure, a graphical indicator can bedisplayed representing that the medical implement is associated with themedical procedure. For example, a visible green indicator (e.g., a checkmark or green medical implement highlight) may indicate a match for theprocedure. In another example, a medical implement identified as beingassociated with the procedure may be constantly highlighted (e.g.,surrounded in the AR display by a green indicator). On the other hand,if the medical implement is determined to not be associated with thecurrent procedure then a negative visual indication may also bedisplayed. For example, a flashing red “X” or a red highlight outlinemay be a negative indicator.

The method may include displaying, in the augmented reality display,medical information associated with the medical implement. Theinformation associated with the medical implement may includeinstructional information describing use of the medical implement in themedical procedure.

One issue that faces doctors and other medical professionals whenperforming procedures upon a patient is making sure that the correctmedical implements are being used on the correct patient anatomy. If thewrong person, the wrong appendage, the wrong location, the wrongimplant, is being operated or the wrong implant, wrong instrument size,or wrong medical instrument is being utilized, then a poor medicaloutcome may be the result. The present technology may provide animproved medical outcome by utilizing optical codes attached to medicalimplements.

In an additional configuration, the present technology may be used for asimulation of a medical procedure. The patient's body or patient'sanatomy may be simulated using simulation structures. The simulationstructures may be plastic or cadaver bones covered with soft material(plastics and rubbers to represent tissues, arteries, etc.) or othersimulation materials. The simulated anatomy may include an optical codeand image visible code. Then image data set for the patient on which aprocedure is to be performed in the future may be aligned with thesimulated anatomy.

The actual medical implements to be used in the procedure may also beincluded in the simulation and may be in the view of an AR headset.Similarly, a limited portion of the medical implement (e.g., just ahandle of a trocar) may be used and a virtual tool in the simulatedpatient may be viewed. Additional overlays using real time fluoroscopy,ultrasound or other medical images captured in real time may also beused. Thus, the medical professional may perform the same functionsdescribed earlier but as a simulation in order to better understand thechallenges, problems or other issues that may occur in an actualprocedure that is planned. This simulation may also be used for trainingpurposes.

Similarly, a medical trainee or an ultrasonographer may combinepreviously captured image data sets (e.g., MRIs or CTs images) togetherwith actual ultrasound images being captured in real time for trainingpurposes (as described above). It may be difficult to clearly see whatis displayed in the ultrasound images due to their low spatial andcontrast resolution. This may enable a medical technician to be trainedabout what organs, bones, blood vessels and other tissues appear likeunder ultrasound observation by using images with better spatialresolution and contrast resolution (e.g., MRI, CT, etc.).

FIG. 9b illustrates a method for validating a medical procedure usingoptical codes. The method may include the operation of detecting visualimage data of a portion of a body of a patient and a medical implement,using an optical sensor of an AR headset, as in block 920.

One or more optical codes visibly displayed on the body of the patientand on the medical implement may be identified, as in block 922. One ormore optical codes on the body of the patient are in a fixed positionrelative to an image visible marker (as described earlier). An imagedata set may be aligned with the body of the patient using a known fixedposition of the image visible marker with reference to the one or moreoptical codes on the body of the patient, as in block 924.

The optical codes and image visible markers may be used to confirm thata correct patient is in the medical procedure based on a correctalignment of a surface of the body of the patient aligning with asurface of the image data set, as in block 926. The surface of the bodyof the patient and the surface of the image data set may be createdusing a polygonal mesh, splines, or another mathematical model for thesurface. If the surfaces (e.g., meshes) are similar or match, then thecorrect patient is recognized. This may be analogized to a contour typeof “finger print” of the body because every individual body is unique.In addition, the identity of the person or patient in the medicalprocedure may also be confirmed using one or more of the optical codes.An additional function or aspect of this validation is to confirm acorrect portion of the body and a correct medical implement are used inthe medical procedure using one or more optical codes, as in block 928.

In a further configuration, the same optical code used during aligningthe image data may be used to automatically retrieve the image data setand ensure that the image data retrieved by the AR headset matches thepatient being viewed through the AR headset without time consuming,cumbersome, and inaccurate manual retrieval of image data.

FIG. 10 illustrates an example system that may be employed to referencea medical implement to an image data set and a patient's body usingoptical codes, as viewed through the AR headset. The system 1000 mayinclude a camera device 1002, an augmented reality system 1020, adisplay device 1030, and a plurality of databases. The system 1000 mayalso include one or more processors, a memory, a file system, acommunication unit, an operating system, and a user interface, which maybe communicatively coupled together. In some embodiments, the system1000 may be, for example, a desktop computer, a server computer, alaptop computer, a smartphone, a tablet computer, an embedded computer,an AR headset, a VR headset, etc.

The camera device 1002 can be configured to capture visible data. In oneexample, the camera device 1002 may be used to capture visible dataduring a medical procedure. The visible data captured by the cameradevice 1002 may include images of a body of a person (or a portion of abody) and one or more medical implements (e.g., medical instruments,implants, and so on). The camera device 1002 may transmit the capturedoptical data to the augmented reality system 1020. The system also mayinclude surface sensors, optical sensors, infrared sensors, Lidarsensors or other sensors to detect and assist with mapping a real viewor actual view detected by the AR system. Any object or surface may bedetected for an operating theater, a patient, a room, physical geometry,medical implements, or any other physical surroundings or objects.

The augmented reality system 1020 may include an image processing engine1022, a reference and alignment module 1024, an image generation module1026, and an augmented display buffer 1028. For example, the imageprocessing engine 1022 receives the captured visible image data from thecamera device 1002 and analyzes the visible image data to identify oneor more optical codes, objects or people in the visible image data. Aplurality of different techniques may be used to identify medicalimplements within the visible image data including but not limited tofeature extraction, segmentation, and/or object detection.

The image processing engine 1022 also identifies optical codes that maybe affixed to both bodies of patients within the image and medicalimplements within the visible image data. Once the image processingengine 1022 identifies an optical code (e.g., an AprilTag, a bar code, aQR code, and so on) the image processing unit 1022 accesses the opticalcode database 1046 to retrieve information associated with the opticalcode. In some examples, the optical code is associated with a particularpatient, a particular procedure, or a particular object. The opticalcodes may be used to more accurately identify the position andorientation of the medical implement, a body or a fluoroscopic device.

In some embodiments, the reference and alignment module 1024 engageswith the image processing engine 1022 to reference any identifiedmedical implements, a body of a person and an image data set withrespect to each other. In addition, the reference and alignment module1024 can use optical code information in the medical implement database1044 to properly identify the size and shape of the medical implements.Once the position and orientation of the medical implement and the bodyof the patient are determined relative to each other, the reference andalignment controller 1026 can align any associated radiology images inthe radiology image data 1042 with both the body of the patient. In someexamples, the radiology images are received from a radiology imagedatabase 1042 based on patient records in a patient record database1040.

The image generation module 1026 can generate graphical data, virtualtools, a 3D surgical tract, 3D colorization or shading of a mass ororgan, or highlighting of a mass, organ or target to display in adisplay device 1030 as layered on top of the body of the patient or amedical implement. In some examples, this information can be loaded intoan augmented display buffer 1028. This information can then betransmitted to a display device 1030 for display to a user.

In one example, the patient database 1040 includes a plurality ofpatient records. Each patient record can include one or more medicalprocedures to be performed on a patient. The patient records may alsoinclude notes, instructions or plans for a medical procedure. A patientrecord can also be associated with one or more radiology images in theradiology image database 1042. In some examples, the radiological imagesinclude a representation of the image visible marker that allows thereference and alignment module 1026 to properly align the image data setwith the body of a patient using the fixed position of an optical codewith respect to the image visible marker. In some examples, the medicalimplement data 1044 includes information describing the medicalimplements, including medical instruments, implants, and other objects.

In some embodiments, the augmented reality system may be located on aserver and may be any computer system capable of functioning inconnection with an AR headset or display device 1030. In someembodiments, the server may be configured to communicate via a computernetwork with the AR headset in order to convey image data to, or receivedata from, the AR headset.

FIG. 11 illustrates a computing device 1110 on which modules of thistechnology may execute. A computing device 1110 is illustrated on whicha high level example of the technology may be executed. The computingdevice 1110 may include one or more processors 1112 that are incommunication with memory devices 1112. The computing device may includea local communication interface 1118 for the components in the computingdevice. For example, the local communication interface may be a localdata bus and/or any related address or control busses as may be desired.

The memory device 1112 may contain modules 1124 that are executable bythe processor(s) 1112 and data for the modules 1124. The modules 1124may execute the functions described earlier. A data store 1122 may alsobe located in the memory device 1112 for storing data related to themodules 1124 and other applications along with an operating system thatis executable by the processor(s) 1112.

Other applications may also be stored in the memory device 1112 and maybe executable by the processor(s) 1112. Components or modules discussedin this description that may be implemented in the form of softwareusing high programming level languages that are compiled, interpreted orexecuted using a hybrid of the methods.

The computing device may also have access to I/O (input/output) devices1114 that are usable by the computing devices. An example of an I/Odevice is a display screen that is available to display output from thecomputing devices. Other known I/O device may be used with the computingdevice as desired. Networking devices 1116 and similar communicationdevices may be included in the computing device. The networking devices1116 may be wired or wireless networking devices that connect to theinternet, a LAN, WAN, or other computing network.

The components or modules that are shown as being stored in the memorydevice 1112 may be executed by the processor 1112. The term “executable”may mean a program file that is in a form that may be executed by aprocessor 1112. For example, a program in a higher level language may becompiled into machine code in a format that may be loaded into a randomaccess portion of the memory device 1112 and executed by the processor1112, or source code may be loaded by another executable program andinterpreted to generate instructions in a random access portion of thememory to be executed by a processor. The executable program may bestored in any portion or component of the memory device 1112. Forexample, the memory device 1112 may be random access memory (RAM), readonly memory (ROM), flash memory, a solid state drive, memory card, ahard drive, optical disk, floppy disk, magnetic tape, or any othermemory components.

The processor 1112 may represent multiple processors and the memory 1112may represent multiple memory units that operate in parallel to theprocessing circuits. This may provide parallel processing channels forthe processes and data in the system. The local interface 1118 may beused as a network to facilitate communication between any of themultiple processors and multiple memories. The local interface 1118 mayuse additional systems designed for coordinating communication such asload balancing, bulk data transfer, and similar systems.

Some of the functional units described in this specification have beenlabeled as modules, in order to more particularly emphasize theirimplementation independence. For example, a module may be implemented asa hardware circuit comprising custom VLSI circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A module may also be implemented in programmablehardware devices such as field programmable gate arrays, programmablearray logic, programmable logic devices or the like.

Modules may also be implemented in software for execution by varioustypes of processors. An identified module of executable code may, forinstance, comprise one or more blocks of computer instructions, whichmay be organized as an object, procedure, or function. Nevertheless, theexecutables of an identified module need not be physically locatedtogether, but may comprise disparate instructions stored in differentlocations which comprise the module and achieve the stated purpose forthe module when joined logically together.

Indeed, a module of executable code may be a single instruction, or manyinstructions, and may even be distributed over several different codesegments, among different programs, and across several memory devices.Similarly, operational data may be identified and illustrated hereinwithin modules, and may be embodied in any suitable form and organizedwithin any suitable type of data structure. The operational data may becollected as a single data set, or may be distributed over differentlocations including over different storage devices. The modules may bepassive or active, including agents operable to perform desiredfunctions.

The technology described here can also be stored on a computer readablestorage medium that includes volatile and non-volatile, removable andnon-removable media implemented with any technology for the storage ofinformation such as computer readable instructions, data structures,program modules, or other data. Computer readable storage media include,but is not limited to, RAM, ROM, EEPROM, flash memory or other memorytechnology, CD-ROM, digital versatile disks (DVD) or other opticalstorage, magnetic cassettes, magnetic tapes, magnetic disk storage orother magnetic storage devices, or any other computer storage mediumwhich can be used to store the desired information and describedtechnology.

The devices described herein may also contain communication connectionsor networking apparatus and networking connections that allow thedevices to communicate with other devices. Communication connections arean example of communication media. Communication media typicallyembodies computer readable instructions, data structures, programmodules and other data in a modulated data signal such as a carrier waveor other transport mechanism and includes any information deliverymedia. A “modulated data signal” means a signal that has one or more ofits characteristics set or changed in such a manner as to encodeinformation in the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media such as acoustic, radiofrequency, infrared, and other wireless media. The term computerreadable media as used herein includes communication media.

Reference was made to the examples illustrated in the drawings, andspecific language was used herein to describe the same. It willnevertheless be understood that no limitation of the scope of thetechnology is thereby intended. Alterations and further modifications ofthe features illustrated herein, and additional applications of theexamples as illustrated herein, which would occur to one skilled in therelevant art and having possession of this disclosure, are to beconsidered within the scope of the description.

Furthermore, the described features, structures, or characteristics maybe combined in any suitable manner in one or more examples. In thepreceding description, numerous specific details were provided, such asexamples of various configurations to provide a thorough understandingof examples of the described technology. One skilled in the relevant artwill recognize, however, that the technology can be practiced withoutone or more of the specific details, or with other methods, components,devices, etc. In other instances, well-known structures or operationsare not shown or described in detail to avoid obscuring aspects of thetechnology.

Although the subject matter has been described in language specific tostructural features and/or operations, it is to be understood that thesubject matter defined in the appended claims is not necessarily limitedto the specific features and operations described above. Rather, thespecific features and acts described above are disclosed as exampleforms of implementing the claims. Numerous modifications and alternativearrangements can be devised without departing from the spirit and scopeof the described technology.

The invention claimed is:
 1. A method for using an augmented reality(AR) headset to co-localize an image data set and a medical implement,comprising: detecting visual image data of a portion of a body of aperson and the medical implement using an optical sensor of the ARheadset; identifying one or more optical codes associated with the bodyof the person and the medical implement, wherein an optical code on abody of the person is located in a fixed position relative to an imagevisible marker; confirming that the medical implement is assigned to beused in a medical procedure using the one or more optical codes;aligning the image data set with the body of the person using the one ormore optical codes on the body of the person as viewed through the ARheadset and using the fixed position of the image visible marker withrespect to the optical code as referenced to a representation of theimage visible marker in the image data set; and determining a positionof the medical implement with respect to the body of the person usingthe one or more optical codes on the medical implement and the body ofthe person to enable the medical implement to be referenced to the imagedata set and the body of the person, as viewed through the AR headset.2. The method of claim 1, further comprising providing a visualindicator, a virtual tool, a virtual guidance system, a virtualhighlight, a virtual procedural track, a graphical highlight of targetedanatomy, a graphical highlight of pathology, a graphical highlight ofcritical structures to avoid, or annotations on the image data set toguide positioning and orientation of the medical implement with respectto the body of a patient using the AR headset.
 3. The method of claim 1,further comprising: retrieving a patient's information using an opticalcode associated with the portion of the body of the person; anddisplaying the patient's information through the AR headset to validatea correct patient and correct body part is receiving a medicalprocedure.
 4. The method of claim 1, further comprising displaying agraphical indicator to a medical professional representing whether themedical implement is associated with the medical procedure.
 5. Themethod of claim 1, further comprising displaying a graphical virtualimplement to allow alignment of a visible implement and virtualimplement.
 6. The method of claim 1, further comprising retrievingmedical information associated with the medical implement that includesinstructional information describing use of the medical implement in amedical procedure.
 7. The method of claim 1, wherein the medicalimplement is a medical instrument, a trocar, a catheter, a needle,orthopedic hardware, or a surgical implant.
 8. The method of claim 1,further comprising mapping a contour of the medical implement using theone or more optical codes on the medical implement as a startingreference point.
 9. The method of claim 1, further comprising retrievingthe image data set automatically for the body of the person using theone or more optical codes on the body of the person.
 10. A method forusing an augmented reality (AR) display to align a fluoroscopic imagewith respect to a body of a person and an image projection from an imagedata set, comprising: detecting visual image data of a portion of thebody of a person and a fluoroscopic device, which is mobile with respectto the body of the person, using an optical sensor of the AR display;identifying one or more optical codes on the body of the person and onthe fluoroscopic device, wherein one or more optical codes on the bodyof the person have a fixed position relative to an image visible marker;aligning an image data set of the body of the person using the fixedposition of the image visible marker relative to the one or more opticalcodes on the body of the person as viewed through the AR display;determining a position and orientation of the fluoroscopic device withrespect to the body of the person using the one or more optical codes onthe fluoroscopic device; generating an image projection of the imagedata set based in part on the position and orientation of thefluoroscopic device; displaying the image projection through the ARdisplay; and displaying a fluoroscopic image from the fluoroscopicdevice that is aligned with the body of the person and the imageprojection based on the image visible marker or the position andorientation of the fluoroscopic device, using the AR display.
 11. Themethod as in claim 10, further comprising: detecting a change in theposition and orientation of a fluoroscopic device with respect to thebody of the person; and modifying the image projection and fluoroscopicimage position and orientation as defined by the change in position andorientation of the fluoroscopic device.
 12. The method as in claim 10,wherein adjusting a position of the image projection, as viewed by theAR display, is based on a change in position and orientation of thefluoroscopic device as detected using the one or more optical codes onthe fluoroscopic device as compared to a position and orientation of thebody of a patient.
 13. The method as in claim 10, further comprising:receiving a zoom value from the fluoroscopic device that has been zoomedwith respect to the body of the person; and adjusting the image data setas defined by a zoom value of the fluoroscopic device.
 14. The method asin claim 13, wherein adjusting the image data set is performed based onusing a size of the image visible marker captured in the fluoroscopicimage to enable the image data set to match a zoom of the fluoroscopicimage.
 15. The method as in claim 10, wherein a fluoroscopically visibleobject in the fluoroscopic image is able to be viewed and guided by amedical professional with respect to the image data set aligned with thebody of the person.
 16. The method of claim 10, further comprisingproviding graphical indicators, virtual tools, or a virtual targetingsystem on the image data set to guide the position and orientation of afluoroscopically visible object with respect to the body of the personand the image data set, as viewed using the AR display.
 17. The methodof claim 10, wherein determining an orientation of the fluoroscopedevice further comprises determining a position and orientation of thefluoroscope device with respect to the body of the person.
 18. Themethod of claim 10, further comprising reconstructing the imageprojection of the image data set and moving the fluoroscopic image, asviewed in the AR display, corresponding to a change in orientation andposition of the fluoroscope device.
 19. The method of claim 10, whereina transparency of the fluoroscopic image aligned with the image data setmay be modified.
 20. The method as in claim 10, further comprising:identifying one or more additional optical codes visibly displayed on amedical implement; and determining a position of the medical implementwith respect to the body of the person and the image projection usingthe one or more additional optical codes on the medical implement andthe one or more optical codes on the body of the person, to enable themedical implement to be referenced to the image data, as viewed throughthe AR display.
 21. A method for using an augmented reality (AR) displayto align a fluoroscopic image with respect to a body of a person thathas one or more optical codes by using a position and orientation of afluoroscopic device that has one or more optical codes, comprising:identifying the one or more optical codes on the body of the person andon the fluoroscopic device, which is mobile with respect to the body ofthe person, using an optical sensor of the AR display; determining theposition and orientation of the fluoroscopic device with respect to thebody of the person using the one or more optical codes on thefluoroscopic device; and displaying, using the AR display, afluoroscopic image from the fluoroscopic device aligned with the body ofthe person by referencing the optical codes on the body of the personand the position and orientation of the fluoroscopic device.
 22. Themethod as in claim 21, further comprising: identifying one or moreoptical codes on the body of the person which have a fixed positionrelative to an image visible marker; aligning an image data set of thebody of the person using the fixed position of the image visible markerrelative to the one or more optical codes on the body of the person, asviewed through the AR display; and displaying the image data set throughthe AR display as aligned with the body of the person and with thefluoroscopic image.
 23. The method as in claim 22, further comprising:generating an image projection of the image data set based on theposition and orientation of the fluoroscopic device; and displaying theimage projection through the AR display, as aligned with the body of theperson, with the fluoroscopic image.
 24. The method as in claim 23,wherein the image projection may be reconstructed based on movement ofthe fluoroscopic device to match a changed position of the fluoroscopicdevice.
 25. The method as in claim 21, further comprising generating animage projection of an image data set based on the position andorientation of a viewer who is using an AR headset.
 26. A method forusing an augmented reality (AR) display to align an ultrasonic imagewith respect to a body of a person, comprising: detecting visual imagedata of a portion of the body of a person and a ultrasonic transducerusing an optical sensor of the AR display; identifying one or moreoptical codes on the body of the person and on the ultrasonictransducer; determining a position and orientation of the ultrasonictransducer with respect to the body of the person using the one or moreoptical codes on the ultrasonic transducer; and displaying an ultrasonicimage from the ultrasonic transducer that is aligned with the body ofthe person by referencing the one or more optical codes on the body ofthe person and the position and orientation of the ultrasonictransducer, using the AR display.
 27. A method for validating a medicalprocedure using optical codes: detecting visual image data of a portionof a body of a patient and a medical implement, using an optical sensorof an AR headset; identifying one or more optical codes visiblydisplayed on the body of the patient and on the medical implement,wherein the one or more optical codes on the body of the patient are ina fixed position relative to an image visible marker; aligning an imagedata set with the body of the patient using a known fixed position ofthe image visible marker with reference to the one or more optical codeson the body of the patient; confirming that a correct patient is in themedical procedure based on a correct alignment of a surface of the bodyof the patient aligning with a surface of the image data set; andconfirming that a correct portion of the body and a correct medicalimplement are in the medical procedure using the one or more opticalcodes.
 28. The method of claim 27, further comprising retrieving patientdata for display in the AR headset using the one or more optical codes.29. The method of claim 27, further comprising providing annotations onthe image data set to guide position and orientation of the medicalimplement with respect to the body of the patient and the image data setusing the AR headset.